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As part of. the national Head Start Planned Variation 
stady; this study used a relatively saall sasple in anfintensive 
evaluation of prograiTiiipleaenUtiaii in' one field oonaunlty. using the 
Tucson Barly Education Modsl rFBSq • % Modified Solonon foor>group 
-research design forasd the organisation fraaieiMrk. Evaluation of six 
TEBI classroom and tso locally iapleimitsd Coaqparison classrooM 
inelndsd a cfhild data bsttery, classrooii obssrvation, situational 
tasks, Oirector«^s ratings, and desoi^aphic inforsation. The child 
data battery, which eapbasised both the cognitive and socio*affective 
doBMins, wss given in the fall of If 71 to approxiaately half the TEEM 
children and to one of the conpaxiaoln classes, spring testing 
inelndsd all grovps. AnSlysis indica^^ed that . conparlson groips were < 
not optinally natcbed to TBBI childnlnt Coiqparisoo children were 
slightly older sod perfomed better on pre-test. The difference in 
levels of inst^pKopriate behavior during teacher absence between TEEM 
and cosparison dassroons was significant. Analysis of classroon 
observation indicated that TBBI classtoons had nore child-initiated 
lesrniag segoesces oanbinsd with lowsr teacher initiated learning 
ssquenoss« Dos to lijpited sanple sise in this study, wide-ranging 
gensnUisations ars not justifisd^ But the utility of eisssroon 
observation techniques for assessing process goals trithin the open 
classroon franework was indicated. |Author/KN') 
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ABSIBACT 



/ 

As pirt of the 'TiationaJ Head Stait Planned testing mclyU^ the loiat TB£M group and both 
Variation Sfudy. this study jscd a relatively srriill CompartHm/ classes Analysis indicated that the 
sample in an intensive evaluation of progcam ' CompaM^ groups were not optintaily matched to 
imi^mentatton in one (kid community using the the j4,0ll children, and differencts in both zj^t 
Tucson Eariy Educatioi] Model (TEEM). A and^^;^ -treatment variables wen^ evident with 
modiried Soiomofi four-group research design (^wt^usoti chtldien being slightly^ bider. and 
formed «Um organizational fran^eworfc for this informing better on pretest > On 70% of setocied 
study.EviluationofsixTEEMciasnoonisandtwo / {subtests from the MiCarthy Scale x>f Children's 
locally 'implemented ComptJDbon* ctassiooim/ ^ Abilities, the TEEM children achieved a gr^sater 
included ^ a child data batttiy (lelecthksubtesi^ raw score incr«aae from pre- tb^postteMing thara 
tfom tho McCaitHy Scale of Children's Abiliiifes. ^ did the Comparison c^dr^n. On two of these 
and the Schaeffer Behavior Invenicwy), cla$«6om jubtrsts, the difference in gains between the two 
observation (Classroom Observation ProcoduteK poupi was statistical^ stgnificant (pc05), A 
situational tasks (Classroom Attitude ONfMtiort onk^r contlation between a rating of the 
Schedule), Director's miinga. and dtafr^ teachers on their impiementaHoa of TEEM and 
information (Head Start Classtoon/hffotmation their pupib* coyiitive ydns as measured by the 
Vom).* ; I : McCarthy revealed the existence j>f a lelationsiup 

Rather than wlying on global tyfe scores and ,bit,iwen these two factor* (r ^ ,63 between 

implementation rating and child outcome). 

The Classroom Attitude ObMrrvattrmSd^^ule 
data disclosed very tittle inappropriate beha^^r 
during teacher absent phae in the TEEM 
classrooms. The difference tn levels of 
inappropriate l^duvfor during teacher absent phase 
between TEEM classroomtt and Comparison 
clasmonu was statistically stgnifkant tp<.OS). In 
the classroom rated as hi^iest*imptemented (from 
director's ratings) the teacher participate with 
children in cognithely-orxnted learning sequences, 
in a classroom rated as low-implemented, the 
teacher engaged in manigement r^ivities* while 
children were mainly engaged in ptayonented 
leanitng sequencei^. 

* The Classroom Observation PnKedure was 
utilized in the TEEM and Comparison classrooms 
to gather information on 4he nature of mieractions 
between teachers and pupils. The variable!^ were 
combined kito eight variable constructs. The 
analysis indicated that TEEM classrooms bad more 
child initiated leamiog lequeiices combined with 
lower aeacher InitfatetJ learning sequences^ when 
compared to the oppoflte pattern in Comparison 
clasmoms. . 

Due to limited sample size in this study, 
wide-nmging gener&UiAtions are not justified. 
Howevei\ the utility classroom observation 
techniques foj^^asSessinR process goals within the 
open ^x^hfssreom framework was certainly 
W\ to approximately half of the TEEM children,^ ^^ftidrtSatcd. Future research will be directed to the 
and to one of the two Comparison cliisaes. Sptftfg perfection of these techni ques. 



emphasizing solely the cognittve'devetopment of 
the children, the child data baUery emphasized 
both .the cognitive and the sopmifrective domains 
of development. The use of nbtesu allowed for 
the testing of progranHq)ecifi/goali. * > 

The subtesu t^'d wew from the McCarthy 
Scale of Cmircn's AbiU>l^. a recently developed 
test released /this year by the P^thologM 
Corporation; The Schaeffer Behavior Inventory 
enabled the cUssroom teadiers to rate their pupils 
on* three/ basic child classroom behaviors: Task 
Orientation* Extraversion andtfostility. 

Th^ Classroom Attitude Observation Scbed^ 
includes categoncs for classroom activities, 
grouping strategies, and tl^ occurrence ^of 
inappropriate behavior m children. This 
instrtmient provides a picture of the children's 
self-direction by recording child behavior in the 
absence of a controlling adult. 

The Classroom Observation Procedure* under 
development by Stanford Researc)^ Institute, 
includes a classroom check list« a verbal snapshot 
of the classroom grouping and activities, and a 
detailed necordmg of interactions. This allows the 
assessment of successful implementation of 
classroom processes within instructional mdSiels. 
Complete data analyses on nhis instrument were 
not possible due to the paucity of normative^ata 
piesently available from SRI . 

The child, data batteiy was given in the Fall of 
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- FORWARP 

The Arizona .Center lor Educational Resean:h and 
Oevelopmenf is a reorganized . unit- within the College of 
Education, Univer^ty of Arizona, h Tucson. Historically, the 
m^or tiirust of tbb group itas' be«n th^ development and 
implementation of. new curri^^ inacilces within the field 
of Early ChiWhtfod Education, Under the auspices of Head 
Start and Follow Through .funds, a long-term program of 
research, d^lopment and Held dfelivefy has brought national 
attention to the' l^niviBnity. However, neWly emerging 
changes in national |doriti«^ within public educatio 
indicated, that a br^er btase, ym n^i^ dor the 
diversificatioh of ttmuidi activities withili the College of 
Education at the Univa!^ty. of AriZ9n9l The Arizona Ceor 
-witfi continue, ite' noteworthy woii J»<the field ' ~ 
Chi^ood Eductt^. while c^pto^ new areia o^ 
and . devel^bpmetC both within ^the State of Arizo 
across the odunti^. 
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n^ROOUCTION 

Preschool programs for disadvantaged chtidreri were mlroduccd on a Jarge ?;cale with the ' 
passage of legislation under Title II of the Economic Opportunity Act of 1964. As part of a 
broad4)ased **war on povcrty'% one of the k^y .provisions was the enhancement of early learhmg 
experiences of children from' loW^income families through a program called Head SiarL Either 
than proceed with a small-scale prototype phase, the Head Start program began nationwide tr? the 
summ(;r of 1965; serving over 500,000 children in an eight wee|; summer program, lajetrospccl, 
the know-how for accomplishmg the broad-based goals of the Head Start program was quickly 
found lacking* This fiJ^t summer program assuaged the national conscience, bui many leaders m 
tho fields of thild Welfare, psychology > and education were concerned about the reality of rnectmg 
the goats of the Head Start mandate Since 1965, program efforts have been conti>,Mouyy 
modlged tof bring a focds on more intensive experiences with fewer children, and brmg program 
doctu^entation and evaluation questions to the forefront ^ 

the .study reported here is a result of that latter fociis As a part of the Planned Variation (PV) 
study of Head Start, the Tucson Early Educatiou^^odel (TEEM) was selected as a program model 
for inclusion in a study of nationally disscfihinated darty education programs. The study deals 
with questions reidting to the development of prograf^-speciOc evaluation tasks as applied to one 
ficM cominunity using TEEM in Head Start. Evaluation studies of Head Start in the past twvc 
, typically lis^d largA samples; and post hoc research designs (Cicirellii K%9,BfSseL 1970) Results 
from^ljiesf studies have been equivocal m showing iastin)!. impact on young children^ 
----^ deyehipment ITius, a need was seen for smaller studies which concentrated on program-specific 
research questions. This approach permits a more valid data base, and ultimately, conclusions 
With higher intemai validity. 

The goals of this study are^ 

1 : To develop an observational-evaluation system to test selected protests goals of the TEEM 
proigram. . 

2, To develop a program*vahd set of tasks to assess children *s developmeni in Head Stan 
classrooms using TEEM . 
. ^3. To field test this battery In a ►small study wrtKsix TEEM chssrooms, and two locally 

implemented Comparison classrooms v . ^ 

4* To perform statistical analyses of potential non-directional differeneerbetween these - 
classrooms. 

'^ These goals were met through a combination of instrument development efforts at the Ari/ona 
\ Center, and a pre^nd-post eoliection of data with children in Head Start classrooms m Lincoln. 
\ Nebraska. This research was initiated with the full involvement of the Head Start Director and 

- I staff in Lincoln. This kind of study would not have been possible without the full cooperation of' 
\ t!ie community and the children in the Head Start classrooms. 

1 Tucson Early Education Model^ • 

! ? The Tucson Early Education Model was Initially developed in 1965 as a cooperative project on 
t|ie intellectual developrnent of young Mexican-American children, conducted jointly by the 
Colfe^ of Education, Uiiiversity of Arizqna, and Tucson School District Number L Under the 
direction of Dr Marie Hughes and Jewell Taylor, this project Ibpghi to identify causes of the 
higli dropout rate of childre*^ from the Mexican-American community, and to develop > new 
educational progTum to enhance the educational experiences of these children? The continued 

• development of this program became the focus of the Arizona Center for Early Childhood . 

. Education, In 1968, the Arizona Center was asked to become a sponsor for the Follow Through 

^ ll^This section is adjjplctl freely from a paper entitled **The Tucson Matly Edutation Modcr\ which wa^ 
pj^j^pared as a prognim overview by the staff fespgeially Marie Hugttes. Rai^h A Wetzel, and Ronald W. ' 
Itienderson) of the^Arijtona Center for Early Childhood l^ducalion. A copy of (h^ extended paper is avallahie from 
;ilii© Information Officer, Artaoria Cenfcf forf*ducaiional Research aiid Devcbpment. 1515 Bast. First, Tucson, 
AirUona, 85719 

• * 
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program, which was designed to providt high-quahty early educanoo progfams for Head Start 
graduates. In 1969. the Office of;. Chifd Dcveldpmenl asked ihe Amotia Center to design a 
comptemcnf|rx Head Siart program in a nurober of the communitje^ already using TEEM in 
Follow Tlirough, ' " 

The rationale for TEEM is based on an anderstanjUngvof the sktJfs and abthnes that are 
necessaiy^ to participate in contemporary America^ as v?uH as an appreciation of the varied 
backgrounds Umi children bring to the educatiortal setting. The content and procedures of this 
program are« therefore, based on the definitibn and specification of the following 

L The and attitudes necessary to function In our technical and changing society 

2* The behsvioruj chaiacterisiics which children bring to the educational situation 

3* The nature of the teaming process. : 

The program procedures suggested by these considerations differ significantly from 
conventional curricula and modes of instruction for young children. If the requisite skills are to 
be developed, new program objectives and priv>rities must be established 

The rJi^or otrjectives of the Tucson Early Education Model can fiS classified into, four 
eateries, * . • * ^ 



language Compeience 


In«5^le<tu4l R5» 




Socteui Aiuand Skilh 



FIgJ K OoaUfiucfureoftCEM. 



1. Languid Cdmpctetice; Language competence is one of the rn^or technical skills of the 
culture .to which, the childreii must adapt* Critical information is transmitted principally 
lit veVbai form* ThTsYequ^ an acquaintance with a var^ty of Hnguisttc labels, concepts, 
langua^^^ forms and an'awareness of the function of langua^> 

2. Intellectuai Base; The intellettuat-base is a collection of skills assumed to he necessary^in 
the process of ieamtng? The skills are as yet only partidly understood and defined, and 
are usually not formally . taught* X?* their importance in every learning process is 
becoming increasingly recognised* ^me of the intellectual base skills involve the 
organization of sthnuii in the enyh^dnment, e.g*« ordering events along certayi dimensions 
such as size, color and fonxi. intellectual base skills ore more complex behaviors 
-which are difficult to define: to be aWe to attend* to recall sisnifieant event,<, to be able 
to organize one's behavior toward specific goals, to evaluate alternatives, and to choose, 
to plan and to develop expectations, to be able to discriminate sgnificant and important 
behavioni in others and to imitate* 

3v Motivatlonai Base* By motivatioltal base, we mean a collection of attitudes and 
behavioral characteristics related to productive social invoNement* These include positive 
attitudes toward school and toward the learning process, an appreciation for learning and 
a willingness to persist at learning tasks, and an expectation of success and a willingness 
to chanj^. . - , 

4. Societaf Arts and Skills* Our culture is characterized by a wide range of arts and skills * 
which constitute social interaction, information transmission, and^ scientific advance^ 
Here we classify reading, writing, and mathematical skills as well as ihe social skills of 
cooperation, planning arid democratic process* Althouie^ certain arts and skills have 
traditionally constituted the primary focus qf school curricula, in the Tucson Eariy 
Education Model they are only a portion of the total program^ 
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The purpose of the instnictionai program is fo stnKtiuie about ihe child a leamtng 
environment design^ to promote the development of ihe behaviors defined by the four goal 
areas. Several aspects of the tnstructtonii program and process have been detineated* 

i . IndtvktuaUudonc Based on the pi^mtse that children come to school-wUh diiTerent sets 
of abilities and altitudes, then it is dear that teachers mast individuaUze their teaching 
procedures. It is a characteristic of TEEM cU^^ooim that frequent opportunities are 
provided for one-tcnone adu!t<hiid interactione A variety of behavioral options are 
consiantiy avaitabie to the child, providing opportunities to develop individual skills at 
mdtvidual rates, . ^ 

2 Imitafionc Although imitation h widely recognized as a sogmfuiant process by which 
young children tearn. it is seldom formally incorporated into classroom practice. The 
attention of children is directed toward the important and significant behaviors of 
others, and the> are encouraged and reinforced tor imitating, {mitatton is a particularly 
important process m the acquisition of language. Adults work continuously to model 
efaborated and extended examples of the child's own Ungttage. 

3 Gratification^ Rewardmg and gratifying experiences are clearly crucial elements in the 
Jearning process Reinforcenlent plays ah important role in cla^ropm procedures Every 
effort IS made to ensure that the child experiehciis ft^<juent gratification as a result of his 
behavjof and skil! acc^uiirttioR. 

4 GeneraUzationo It is crudaf to Ihe success of an educational program that the skills which 
It teaches can be extended by the student to .a variety of settings, objects and events. A 
sktll h always taught m a functional setting; and is illustrated by a variety of examples, 
m natural contexts. * • > 

5 Orch^tration, Th^ various skills wWcb reflect the four goal mns are seklom exercised 
;ndependent!ye It is a central aspect of TEEM tliat these skills are . developed 
simultaneously through learning experiences that have interr^la/ed goals. The technique 
of concurrently attending to and developing a variety of skills in children is defined as 
orchcsiratiom ^ ^ 

The major obj^tives of TEEM specify the dcvetopment of skills and attitudes necessary to 
function tn a technical and chan^ng society. It is recognized, however^ that in coping with their 
own environment, children develop other iskiUs and attitudes which are quite functional and 
appropriate in their own homes and neighborhoods* Teachers and aides in the TEEM program 
learn to use the experiential backgrounds of pupils \o further irt5tnictionsl objectives. The child's 
home and neighborhood are viewed as instructional resources, thus avoidmg the discontinuity of 
cultural values wjiich often confront minority children in school. 

Imnlementation of the TEEM program m Head Start depends on a systpm of uaining and 
supp<m services to x^ommunities. The delivery system depends on a multiplier effect, iri which 
field representatives at the Arizona Center train progntm as^stants in the communities, who then 
train teachers and aides at the classroom level. P^nent trainbgstrate^esand research findings* 
tested and detrpstrated at the local demonstration school in Tucsors^ are communicated through 
the delivery system to community staff for implementation in the c5a;^oom setting. 

'j PAST RESEARCH 

I 

: ; » 

A maylor siuh^ of the long term effects of Head Start experience was conducted by Ohio 
University, in oollaboration with the Wcstinghouse Corporation. This report, called The /mpaci 
of Head Start (CiidreUi, 1969}> Was a crosshsectional analysis of the performance of children who 
had a Head Stat;^ experience, and were in the <^rty elementary grades (first tiuough the ihtid 
grade). Contro! gi^oups were formed by matching each Head Start child with a chikl in his current 
class on selected demographic and educational variables* Children from around the country in the 
first, second and tfiird grades were sampled in this study. This kind of diraign skirts the importAnt 
4ask of specifying pe classroom treatments in onier to better understand the pattern of results. 
In the analysis, Head Start children were separated into two groups, one which had a full-year 
'Head Start, and on^ which only. had a summer experience- 

Th<Hevaluatlon measures consisted of the lUinois Test of Psydiolinguistic Abili^^^ 
measures of school readiness and achievement » and projective attitude raeai^ures completed by the 
children. , 

-I . : ' 
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The overaU conclusion was thai no diffwences existed between Head Start children and 
control chUdrcn on any of the above measures. There were iiifTerences between some subgivjups. 
and across national regions, but this did not modify the overaU nuJJ findings. The report 
conciudet that summer programs appear ineffectwe in enhancing cognitive and affective growth 
The report lacked any statement about the impldroent^tion of medflc program mbdels. and 
md»catcd wide vviance among children and groups. It was thus concluded that Head Start 
programs were meffetiive in influencing the development of disadvantaged children into the 
elementary grades, • i 

The rmdingrof the Westinghouse study were controversial, and many researchers criticized the 
design and analysis for being inadequate t«Ji the task (Madow, 1969; Smith & Bissell. li?70). This 
led to an intensive re-analysisof some of these data, in a study by .Smith and fiissell (I9y0). They 
modified the analysis plan with aew covariance procedures, and focused on first grade s<Ares with 
urban black children. They concluded that the Head Start program was succcssfuHWth this 
specific sub^oup from the larger sample. They also noted that the most serious probleki in the 
design of the Westinghouse istudy was the goal of assessing the "overaU effectiveness" ^f Heaa 
Start through massive sampling of graduates. They called for, as others had before ihemvtemallet 
studies which used quasi-experimental designs. 

These criticisms of past research led to the qjedficatiori of the lon^tudinal Head Start planned 
Variation research design that is currently being conducted nationwide to assess the relative 
effectiveness of eight preschool program models on Head Stan children. famiUcs and 
commumties. This longitudinal study was.m?>deled after the Follow Through program, and the 
Ofiice of Child Development contracted, with the Stanford Research Institute to coUect da»a to 
evaluate all aspects of Head Start PV irnjlementation. The first report on this study was 
produced irf 1972, under thedirectionof Joah BIssell. In summariring themassof dati collected 
on children, teachers and parents, it was noted that all Head Start programs, Iwth model and 
tegular classes produced gaiiis in cognitive development and ?erteral achievement that were 
larger than could be expected by maluiation alone. It was also noted that in the area of cognitive 
development. chUdren in model programs made greater gains than children in regular classes. 
They note that, with two years left in the Planned Variation study, these findings are indeed 
prelimmaiy . and aw^t later replication with upcc^ming classes of Head Start children, • 

The above studies focused on large, nationwide samples of Head Start children, and a global set 
of measure* Another set of studies on Head Start age children have used small samples of 
cfuldren and Intenswe research designs to answer specific questions about preschool effeciweness 
One of the mostjmportant research efforts in preschool program, development has been the 
elaboration and comparison of competing curriculum models for use with disadvantaged 
preschoolers. The most extensive effort in this direction was a recentiy completed study wWch 
mv^tigated the relative effects of three popular preschool modeU (Weikart, 1972). These three 
models were: 1) a Lai^v^ model, modeled after the woric of Becker-Englemann; 2) a Cognitive 
model, based on the theories of Jean Piaget; and 3) a Unit-Based or traditional nurscry^Aool 
program. Curriculum variaUon was the critical research dimension, and other program 
components such as supervision and home contacts were held constant The children entered at 
thrw years of age, and participated for two school years before entering public school 
kindergarten. All of the children were described as disadvantaged, and aU met the criterion of 
bebig, functionally retarded" at their en&y into preschool. Each classroom had t - teachers, 
and one community aide, and aU homes rece^ed a home visit once every fortnight \ research 
design mcluded the investipUon of the children's development of many measures of cognitive 
and sodo^raotional growth, investigation of classroom teaching styles, and investigation of ocer 
mteraction processes. 

Re^ts of this three year project indicated that aU.chiWren benefited from participation in 
preschool in terms of iiTteilectual and emotional. piowth (Rentfrow, 1971; Weikart, 1972) 
Differfences in teaching style and peer Interaction were minimal, and not significantiy different. 
Weikart (1972) concludes Uiat certain dimensions of program operation are necessary conditions 
for sudossful preschool intervention, indi'pcndect of curriculum. The most important of titese 
are teacher ptonning, and coramittinent and provision of a growtit-oriented model for supervision. 

DiLorenzo and Salter (1968) reported a study of preschool prograhis incorporating eight 
tJifferent school systems assessing the longitiidinal effectiveness of preschool for disadvantaged 
chiWren The children were selected as four year olds, and were randomly cast into treatment and 
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control g;youps l\dcU of the t*ight vtstctm wa^ tree lo develop lU^ own model wuhm the o^efdH 
goah of ihc pro;eci- The specific cumcutum foci ranged from Bccker-Ehglemann to Montc^^vjn 
The evaluation desigrt consisted of spnng testing on standard measures of intellectual 
functioning (StanfonJ Bmct, Illinois Test of Psycholinguisiic AbiliUes. Peabody Picture 
Vocabularji Te$i), and school readiness (Metropolitan Readiness). After the preschool year, the 
* experimental groups made modest gams m IQ <I to 4 poinfsK as compunKJ to slight fos;scs for 
control children Analysing effects over ail prOgrarns, they conclude that "The most tffccti^e 
prektndergarten programs were those with the tnosi speciflc and structured co^ittvc aciivitjes " 
(p^ > 12). At the end of the kmderganen yea^/ihc experimental groups mamiamed supenontv 
over the control groups m terms of school readiness. The data were analyzfd by race and sex a^ 
well, and the non-white expcnmcntaJ children did not maintain their gains after kmdergarten on 
measures of inteRectuaf development Also» the non^whitc experimental children wcfv 
Sigmficanri) lower thaa the while experimenfaf children. on the Metropobtan Test of school 
readme^s 



CURRENT RESEARCH DESIGN 

The previous review of research stiidies on Head Start, and Head Start-like, preschool programs 
provides an illustration of the dnemma that confronts evaluators in designing studies m thifcYtcld 
A research design must bc^ judged^ both in terms of internal vaJidliy and txfemaf validity 
(Campbell & Stanley. 1963). Tfflie former relates to questions about the effectiveness of the 
treatment, m this case; implernentatlon of a classroom program. The latter relates ta the 
gcnerali/abihty of the findingi to other classrooms or other children in retrospects tf|e 
Westinghoua* study was designcjll to maximize external validity: thai is, children from all regions* 
i>f the country, from a variety* of classrooms, were tested on global tasks On the other hand, 
studies such as Wcikart's comparative cunricuium study conantrated on a small sample of 
children, randomly -asagned to classroom models, and tested on more speofic risks rdcvani to 
program goals* This design seeks to maximize internal validity, at the expense ol ^cnerahzahiht^ 
to programs and children all aaoss the country. Campbell and Stanley iretJt these vatidit le?^ as 
trrterdependent: that is, internal validity is a precondition for externa! validity The^' concerns 
fiave obviously influenced the Office of Child Development .(OCD) m its cufteni evalujiion 
program for Planned VariaNon Head Start 

The national Planned Variation study was formulated to test the relative long lecr. cOkac> of 
eight popular preschool models in Head '^'tart.Cfonbach (1963) described this kmd of cvjluat^on 
design as a^'horsc ratce" stiidy The data u)llected by' Stanford Kese^irch Institute tSRl) as part of 
the national evaluation study of PV, on children, teachers and parents can Inr an3!> /-d n tcrrar 
of 'Vm» place, and show'* along different dimensions of Head Start goals. Contunent wah the 
SRI evaluafion, OCD leadership decided to supi>ort small, sponsor-specific studios to help vaiid^jie 
questions Of program implementation and urlpact. Tliesc two kinds of studies then provide 
\ complementary ir^formation in terms of both internal and external validity 

Three different TEEM Follow Through dtcs were chosen for mitial participation m the Head 
Start Planned Variation study. These communities were Lakcwood. Nevj. Jersey • Walke/ County, 
Georgia; and Lincoln, Nebr^a. These communities were selected in terms of having an effective 
^ and continuous feeder system -of Head Start children into Follow Through claises of the same 
. model sponsor A fixed number of children in each immunity (about 1 20 each year) have 
^ participated in Planned Variation TEEM Head Start classrooms for the three year duration o| the 
study. Tlte national evaluation study conducted by SRI has collected information on chiioren, 
C 0 ^^^c^^' parents during these three years: These data are being interfaced with similar 
longitudinal information collected on Follow Through children. Ultimately, three cohorts of 
pmi^ children will have experienced Head Start and FoUpw Through for five years under an integrated 
program model These data will permit important questions to be answe/ed conceminj} the 
relationship between program models and differential outcomes in children. 

After the large national study oCPl^nned Variation was^ mitiated. the need for smaller, sponsor 
specific, studies, was foreseen. Sponsors were a^ed in 19T0 to propose discrete studies'to provide 
valuable information on model-based questions Of the three sites implementing TEEM Head 
Startv one was selected lo provide the maximum amount of information withm a limited budget 



The community selected for thi!» siudy was Lincoln. Ncbfd<ik3 Thii middle v.^eJ ..nnmi;nif> 
w the Greal Pbtns has m^mfdined a ^imng, comr?)unit>-bas<?d commttmcpf lo the luu<^n l^tU 
Hducatton Model stncc iXb mtfoducuon m 1969 The Head Sfan Dirc/fo; hast^jcn mv il%c.J m ?hc 
devctopmtnt of programs for low-mcomc ctuldrtn Lincoln jincc their invcp?;r»n Hh- 
commumty*5 commrtracnt to TEEM has bcvn m wide p^jftictp^jjon m TtFM Tfjming 
«i«ions by stafT from the larger school syslcm The program his al^ tx'cn roieJ ^v an av irvc and 
well accepted parent invo!vemenf componeni The communu> is prt*dOfninjn!l> A/igio \Mtn ^ 
small percental of Negro children ;ip9c in ihi^'^ampie). and vifiuaHy no other cihmc ^ruupj^ • 
The predominant famUy background of- Ikad* Surt chiidr^n. then, of fann^KS ih^t have 
immigrated from the small fanns of the fUms and Southeast Thcs<.* tarmhcs jtv JisLnbuicvJ 
widely acio» the metropolitan area, and there is no core, or ghetto ' area oi poverty In i?«is 
community, during the 197! »72 school year Head Start chaldirn wea* pinid m dasstoom 
locations within elementary schools in the community Thu>. mosi of the chiKucn were bused to 
thc<r Hc^d Start classrooms 

The selection of Lincoln for the study was premised on a htiJ^-wplementat^on site that offered 
full cooperation to the program sponsor m doing evaluation research Stable fmplcmcntatton is 
assumed to be a necessary (but not sufficient) condition for influencfng children's development 
Medley and Milzcl (1963) note that our concern for the implementation of *ducatioiiar systems 
must ultimately be focused in terms of effects on pupils, or changes in pupil t>ehavior With this 
premise, the following study was designed to spetify the goal areas of TEEH Head Start in terms 
of operational definitions, and to use these.domains m cojiiparing^the growth of children m si^ 
TEEM implemented ^Head Stan classes with children in two locally implemented classes Another 
element in the design is the development and validation of a new technique to assess process 
outcomes m TEEM classrooms 

The selection of dependent variable measures was based on the following constderations The 
TEEM model puts an emphasis on children^s development in both the cognitive and Mie 
socio^ffecttve areas. Traditionally, evaluation designs have leaned heavily. on the former ds the 
major outcome area The de«gn outlined for this study l?uts an equal emphasis on both domajns 
Rather th«n depend upon global scores, such -as IQ, this design depends upon the specification of 
a se! of tasks which ha^u face validity m the TEEM program to e^-aluatc a specified sei of 
outcomes. Also, the de^gn is explicitly child^based. since a wide data base ccnceming teacfier 
training and rmplementatftm is emanating from The TEEM Follow Through program 

The selected depepdeni v-anable measures, grouped by goal.arcas. are given be^o^r 

!nteaectuai Skills 
. Pictorial Memory - MSCA^ 

Verbal Memory 

Conceptual Grouping ' 
Motivational Base 

Schaeffer Behavior Inventory 
Societal Aits and SkDls 

Counting and Sorting 

Imitative Actmg 

Drawing 

Number Questions 
Langii^ Competence 
Word Knowledge 
Verbal Fluency 
Opposite Analoijies 



"T>tesc comment? arc b^^d on mfomjjtion gifttcfed on tht fk^J Sta^r Ck^iroom Irtfaryr^tfon fof^ wh>lh 
swfnfT.a/wcd infofnuiion on n-jnoa^ (4iu*l> allnbute^. luch asuiccmc. pairnf*^ education, pijtnt i pUce of ^/rth 
and vocational lusiory 

•t 

^AU of tbe wbttsts. except the Schaefcf Behivio? Icvcniory . aic from \hc McCa/thy Scile of rhiidrcn's 
Abditics TbiA test wij uliiued tn pie-pubhshed form thiougis an^angerocnis made wjth Dt Alan Kaufmin. 
Resc^ich fV>Tfaok)gtst i? tht F$ycbo}og3ca) Corporation Staff at the Amona Ccnrer had been invoUfd ?hc 
e«fly itaitdardizitjoo of rhe McCarthy, and fell it «rai a ptomim% ntrW djfrctton lor as*e!tf??ng doclopmcnt tn 
youtia chddxrn Psychologjcd Cofpor^lfon ctpccis lo rcltaj^ tht f«t pubUcly by Srptcmb^r 19^7 (p^rx^nal 
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Inr ^ r-UJ ^'iv ij 6,t\s set uuUk^ Cbtc^ts fiom the McCa-^thv Sca^c of Children's Abihircs Ttii^ 
^^? fvvcnti> aovljpcd by the Psychologic<i! CorpOf3t<on. rcprrscnls the tulmtnauon ot the 
w^ofK i)t r><vot! -J M;.€afihy. ? prominctii chili ps>chologist TTic Scak iifilacs both paper aud 
;vjr.:? jnd ro> MKc fP4ten4l^ m ihe mdtvnluaily/sdmmtstercd as&e^t?nt of imcncitual and 
?H^»5jviora! dcsclopmcn! ifdrcn JVr to 8^: yckn of dgc 'The goats ^htch gijidcd tot 

development were to make the suWests (a) intnnsicdHy inicrestrng to yoar^g chiWren. 
•.motionaUv ntuira!. and (c> tclaijvtly qwcV to admtnisier in order to pcrma the S3mplmg of a . 
wide vanctv of cM^Mviors and avoid taxmg the shQtt allcntjon span of children. The sckctlon of 
^tibfesis for thi:> smdy was based on the need to c^pljcaic spcafic performance Remains r^ithe? 
than depend on lota! lest scores The nantbcr of ttcm% per subtest v^nes. but generally a pattern of 
incfL-aMngJy dtfficalt items is followed. Gustered by the TBEM-^aficd goai arca^.thr-sub-e^^. 
employed m this Sludy are further expSitated in Figure 1-2 ' 

Children from culturally different backgrounds otten face the educational environment w»th a 
negative perception of seff-worth. One of the intended itpp^cr^rcas of TEEM is t6 improve this 
self concept as a facthlattng condition to other learning expenchces Ttn: Preschool Scl(^oncepr 
he we Test (PSPT) was developed byAVoIlncr for use with preschool chiWren in Hea* Start site- 
to assc^^ f he atniucjes that children have tov^ard themselves The test format contains equivalent 
picture stimuli for both Caucasian and Negro children, and forms for boys and girK Th;: ptciurv 
stimuli represent ten criterion attitifdmal dimensions, gresented in bipolar fashion leg 
dirty^lcan. independent nJependcnt) Tltc child selects the one which he feels repnrsents hm^scii 
This technique ytelds ^ seJf-pontivity score> which may be analyzed in 3.pn>posi design 

The ^pc\too\ Self-Concept Picture Test, initially yitendcd for use in asses^^mcnt of mot^v^uonaf 
base, waidisearded for further use after pretest analysts Scored for positisity of self<oncep! tu 
uncorrected spht half reliability estimate was • 065 Choices score 1 aJ '^positive" such as ' Mfo-i,? 
for boys and' '^wcak^* for girls lack even face validity m many cases, and the null >phi ha!5 
(consistency reveals that there 's no construct » such as postivity of ^Jf-concepl. (iridcfivmjii ii- 
items It IS doubtful whether four year-olds are (Jcfmmg sclf-viiv. m tcrmv the a lca^r 
attributes when asked to choose between two pictures by the phrase '^^Ttich^ov are vctP Tru 
IS further revealed by the fact that when asked* "Which boy would you to ho'" ndcji 
>:ff-<onc<?pt) the children chose the so^called '^negative* pkXmx^ more f<cMU(:ni!> thjn ^hx^^ 
defining actual *se!f-concept on over half of the aitrrhutes (eg . 67*5? of Kiv -.hihlfcn jjMad 
themscK-es as happy by miUal picture choice-, bu? only 6lOf supposedly witrhtj.j to tx* i'jpp% ^h'.n 
maktJig the picture choice* for ideal ^eff-cortcept) 

If was decided fo substitute the Schaeffer Behavior Inventory (SBI> um the PSPT Ui 3 > 
motivational base growth m children The SBI, as adapted by Stanford Rcscar Ji In^hM;?-. k 1 
fifteen item checklist designed 'o assi<?t teachers m assigning scores to three ba-u. ;.bj!j ji- ro^ n 
t>ehav lors Tliesc' three behaviors are 

1 Task Onentation - now well a child attend^ to and slays with l-^^yoom actuincs 

2 Extraverston - how readily a child mtrr^^cts with other people 

3 Hostility - how a child responds to some of the adjustments and conflict probkmb 
encounierrd in group activities 

The scale used for each item fs of the Ltken-typC; with seven options never, almost ne^cr 
occasionally, half the time, frequently* almost ^*vays, always. For data analysis, aval* ofonv 
assigned to '*nc\er** and a value of severe to ''always*' Under t^ic auspices of the St.^ntorJ 
Research Inststute study, the six TEEM Head Start teachers completed an inventory on vJ^h m 
their students m the early Fall of 1971 Talcing advantage of this available pattest data. 11 f M 
teacheTi were asked during posttcsting to re-rate childncn in each 6f ihesi: clasv,-s Toacher^ tn ?^ c 
Comparison classrooms rated the hildren m Apnt only 

TTie Head Sim ClassfOim Ir imuon Fon/t (HSCIF) ^vas dox-loped by Stanford Hc^afih 
Irrstitute to gather summary ^rmaJion on background vanabtcs mns^dct^ impo:tan? m 
Mnder5tandmg the develo T»ent of children m low-mcome familic^s^ Many siudies have mdit^tul 
that home cnvuonmenl r^ablcs are important predictors or covar^ate:> m undervtandm? ihc 
relaisonshjp between ch .,rcn*s learning and rhetr school context (Hess & Shipman I96S. 
Hender^)n 1972), The HS^.TF asks questions abo5t famuy income level occupational hBtork-^ 
of thr parents, educational backgrounds of the parents, and the presence of arnenities m thv 
hon3e The tofvn is completed by teachers from the Head Start classes. 3;jd typically rvq-jtr. ^ 
home contacts *ith family members These data were colkclcd m Hc^^^ Start clasK-s dur^--j? tu< 



ftiU of I9f Much of the same loformatJon on the Cdmparison classes was collected during the 
Spring of 1972. Since these are essentially stable variables, no bias is expected between these 
groups as a result of different CQltectiO[[i times. For the Comparison group, only parent education, 
family income, and family occupt^iQmwere collected. The description of the Lincohi Head Start 
fiunilies, 'based on these data, was di^scribed eariier (see page 6); 

Another majox goal of this study was the developrhent of a new technique for asse^ng certain 
.^process goals'* within the TEEM Head Start classrqpm* Thes^ process go^ have conststdntly 
eluded fomial evaluation, since young children dou\ ftspond meaningfully to paper-and*pencil 
attitude or selCHX>ncept tests. Woricing Jfrom the Classroom Observation Procedure developed by 
SRI, combined with procedures develc^ped at the Arizona Ceitter, a new technique incorporating 
cla^room observation techniques with ej^vironmental manipulation was dh^eloped. 
^ The Classrobm Attitude ^bseryatidh System (CAOS) includes categories for classroom 
activities, grouping strategies* and tli^ occurrence of ihapj>topriate behavior in children. Ttie 
observation is divided into tKree phases: basettfie pha$3 of ^ minutes; reacWafr5e/t/ phase of 12 
mhiiiteSt in which teacher and aides are excused from the room; and ireinstirution phase of 12 
minutes^ with teacher figures agpv present. It is hypothesized that children in a \*EEM classroom 
develop internalized self-ditection, ift* <Jo^not need the extemaU implied control of teacher 
presence to continue patterns of learning Ibel^avior. 

. J • • ' 

A B - C . . 

* ^ i2min< I2min. 12min. 

il^tt^u present Adults absent • Adults reinstated 



Rg:'f-3. Phases of CAOS. 



TIk qbservaiiona! technique, cour\t& children and-adults engaged in the vanous classroom 
activities on a time^sampie basis. Ali of the types of activities presumed to take .place in the 
classfoom are listed on the recorjling form (Figure Mf Once ev^ry two minutes a clockwise 
visual scanis made of th^ room by the' observer. The observer remains stationary throughout the 
thirty-six minute period* d$ the scan begins ami ends at the"samrpointfor each scan. Numbers of 
children and adutu observed during that scan are placed in the appropriate celU while retaining 
grouping patterns in the recording. If-inapproptiate behavior is observed during this scan, it is also 
noted by its associated activity and4n the appropriate two-minute scan period. Two more scans 
are made during the two-minute period to pick up incidents of inappropriate behavior, once at 
the end ofthe minute^ and again at the end of a minute and a half. 

^ Procetfures and categwes from two previously developed instruments' were ^nthesized to 
pfcKlu(p&. thi^ partieularT' procedure. Both of the parent pwcedures have been field tested and 
founo^liable. / . 

Oni of the two instfumenh'was develQjie^l by Dr. Jane Stallings, of Stanfoni Research 
Institute* ft is an interaction observation t^nique developed for National Head Start and Follow 
Through evaluatiqrj Efforts, called Classroom Observation Procedure, As a preamble to each 
fiY^n^ittute intfwction recording period, the observer takes a ^'snapshot" which gives the 
foundation for Aie'CAOS technique. Activity categories are take l directly from the*'snapshot". 
Within SRPb coding system, grouping patf^rns are classified as for 'MndividuaP'. *'2" for 
"pairs'\ "small groui«r\( three to eight children), and "L" for *1arge groups" (more 

than eight chifdren). Given this system the total number of children engaged in an activity at a 
given time may not be apparent. With CAOS, numbers are used for all groups recorded in 
order to add this dimension to the procedure. Activity definitions used by SRI are more 
encompassing th^n th^ used for CAOS,. in order to include all types oT curricula and all grade * 
levels. involved in* Head Start and Follow Through. While activity deRnitions for CAOS follow 
SRFs system essentially, they have been redefined to make them more program specific to the 
Tucson Early Education Model, and to deal more appropriately with pursuits of four and five 
year old children (Goldupp, 1972). Figure 1-5 is an example of such redefinition. 



; I- 



1. 



1. 



i H 



t 

4 

i 



1 

4 



J L 



i — h 



Li 



i_l_Li 



r}6 



« 7 1 i> 



" i < 

I 



I' 



is 

sis 
5!l 



ill 



It . 

1 5 ll 



-5 i 



t < 

1 1 



E 

< 



10 



SRI Version 

*'This category refers to the teaching and- learning 
about plants, animals, minerals (care of, collection, 
comparison) and science concepts. It is subdivided 
mto texts, virorkbooks; plants, anintals; science 
equipment; and films, slides as tools for teaching and 
learning in this area* A!! of these have been defined 
except for scierKv equipment, whtch rbfe0 to any 
apparatus or conaete objects used In the' course of 
teaching and learning about science and'the natural 
woHd/' 



CAOS Ver«i>n 

■ . ■ . / / 

This category refers to teacjrting and learning about 
plants, animals, mfherall /care of, collection, 
*CQmp«t*son) and science ^&o;4cepts. With very Voung 
children this may consist of^ 

playing with a classroom animal such as a rabbit' 
or gerbil 
* looking at fish in a unk 
looking at objects throuf^ a microscope. 

Included in the science concepts would be the 
intellectual skills uught through a variety of means* 
This M'ould include ''Intellectual kits** whkh are used 
for fo^ering observational skills, and for making 
comparisons, among oUier things. This will also 
inchi^ activities dealing with shapes and the 
disct^mination skills associated. 



Fig. I S. Con^wison of SRI and CAOS definition for one activity catetory : 
V "Science, Nattrnl World**. 



% 

SRI found the snapshot portion of their procedures highly reliable (bener than 909;- over 
thirty trainees in a stationary test situation) and the training of observerstelatively simple. 

The other parent instrument was developed at the Arizona Center for Early Childhood 
Education by Drs. Sadie Grimmett and Billie Underwood. The Schedule for Incompatible 
Learning Behavior (SILB) (Grimmett, Underwood & Brackney, 1^70) was developed for a study 
assessing the relationship of behavior settings to disruptive or inappropriate behavior. With some 
modification, the ^'AOS categories and definitions for inappropriate behavior were borrowed 
from this instniment. The categories i*^nsist of ''hitting'\ **yelUng*\ and ''other'* to replace 
SIlB's ''disturbing**. If a behavior wa*^' observed that was cleariy disruptive but not part of the 
exis^ categories, the observer tx>ded this "other** and noted the behavior at the bottom of the 
obsemtion form. 

The population of behavior sampled was that occurring during fret choice time in all 
classrooms. This is the time when chUdren choose their own activities from those available in' the 
classroom. The decision to standardize vprocedures on free choice timestemmM frpm a 
combination of reasons: / 

a> The behaviors! setting needed to be consistent across ill classrooms. Gcimm^tt et^ai. 
(1970) found powerful evidence that the behavioral setting controls incipience 'of 
disruptive behav^pr. ' . \* , 

b) . Individual choice time was associated- with lower rates of inappropriate behavior than 

large group time (Grimmett et al.. 1970). 

c) The block of time during which children choose their own activities was the ly>ngcst time 
segment in both TEEM and locaHy implemented Head Start Classrooms / 

Observations took place the last week*«f April, 1*^72. with one thirty-six minute observation 
period for each of eight classrooms dujdng^that week. Two ob^ners observed in four classrooms 
each Training consisted of a review of "categories and their definitions, and practice codmg by 
bo'h obser\^ers in the same classroom, without the manipulative phase. After practice coding, the 
two observers met to compare codmg and isolate trouble spots. When definitions were not clear, 
they were re-workcd until both ob^erj could agree on their meaning and obserN-ability. The 
observers achieved rehabiUty. with average agreement of 82"^. and a high of 9!% (Scott*s pi: c.f. 
Flanders. It was felt that this was high enough to permit comparability of obser>ations 

made by the obseners tn separate observations. 




The Classroom Observation Procedure (COP), developed and fieWed by s^ff ft6m Stanford 
Research Institute; provided additional information ojn the operation bf the classrootns within 
this study. Staff from the Arizona Center visited SRI in Palo Alto, CalifbmMv in April, 1^2, and^ 
coded all of the COP protocpis taken <Jn the TEEM clasttocnns during the Sirring of 1972, The 
Arizona Center contracted directly with the SRI train«d observer to observe^ the addwonal 
Comparison dass&rooms in tincob. The GOP collects information on. the physi^ plant, type of 
ongoing chissroom activity and locations of teachei^ and teamers, as well as ongoh^Keachern^ 
interactions. Twenty-four five^inute observation periods comprise the informlition on each 
classroonu The mass of data was reduced thitouj^ computer programs developed ^^ciatly for 
the CW^$/ttm by programniing staff at the Arizona Center. , . \ \ 

T)i€ Head Start Director in Lincofai was asked by SRI as part of their national study\tb rate all 
of /the TEEM classrooms in Lincoln. This rating was done on a ninenitem Likert forn^t, with 
overaU implementation being the variable under scrutiny. The Arizona Center asked the Direct^ 
. to add the two Comparison dassrooms into this format, and rate all of the clas$room$\>h the 
same form. These data comprised the independent ratings of TEEM implementation. \ \ 

The design of the study viras based on a Sotomon 4^up paradigm, which introduces certain 
economies in costs, as well as providing answers to questions aboijjt <hi potential artifact of 
repeat testing on group means. The Sotomon design invoWes pretesting only S0% of the 
treatment and control groups (subjects are randomly selected).^ Posttest is administered to all 
subjects in both groups. This ideal dedgn was oompromised in. the following ways* At pretest 
time/ one total class of Comparison was randomly selected, rather than selecting subjects 
raillomly. Additionally, due to high attrition in chikiren this age, all subjects were posttested, 
whether they had been pretested or not. See Figure 1-6 for graphic design of this study- 
* • *" 

*nic N's gjVcn in Fia. 1-6 tre bmd on the pt^posed operaiion for the Heid Stwt program in Lincoln in 
197 1 •72« Du« to tttritiOQ and tbsenm, the N used Ui the analy«ei^is lightly smaller. 



Child Dau Battery 

McCarthy Scale of Children Abilities 

Word Knowledge 
Counting arid Sorting 
Veft>al Memory 
Drawing 
VerhaJ Fluency 
Opposite AnaK>gies 
Conceptual Grouping 
Pictorial Memory 
imitative Action 
Number Questions 

Schacffcr Behavior Inventory (SRI) 



Ctassroom Observation 

Qassroom Observation Procedure {SR}) 
Situational T>sks ' ^ 

Qassroom Altitude Observation Schedule 
Demographic information 

Head Start Classroom Information Form 
TEEM Imptementation Rating 

director's Rating 





N ' 


Pretest 
Sept., 1971 


November 


Februar.y 


N 


Posttest 
April, 1972 


TEEM Head Surt 


N-60 


Child Data 


DenK)graphic 
Data 


aass 
Observation 


NH20 


Child Dau 


Comparison 


N--12 


\ 

\ 






N-24 


Director's Rating 
Situational Tasks 



Fig. 1-6. Selected instrumentation. 



12 



RESULTS 



In the following discussion, there are three distinct data points «iat be reported. The first 
analysis is concerned with pretest scores only; the second is concerned with pft*toi>ost gain 
scores for children who remained the whole year; the third is a post-onty analysis for all 
children in classes at thb end of the year. The second group, subjects who were te^ed at both 
data points, provides the most important data for analysis. The recder is cautioned to follow 
closely the different levels of analysis as pointed out above, since the power of certain analyses is 
a function of which sample is being reported. 

ReliabiUty 

Sixty-four of th<\chifdren tested irt the Fall of 1 971 were also available for testing the following 
Spring. Test-retest reliability coefficients on the McCarthy subtests and a McCarthy summary 
scortf^ere calculated on this sampfe (Table 11- 1). Thf McCarthy summary score is ii partial total, 
since ibis based on the raw score sum of only the teit subtests which was selected for use in the 
TEEM evaluation from the complete McCarthy Scale of Children's Abilities. Reliability estimates 
on the subtests ranged from .18 to .78^ with a mean,pre-post correlation of .55 across all ten. The 
McCa^hy summary score displayed a test-retest reHabiUty of .87. This represents a considerable 
degree. of stability given the seven-month interval between administrations, and any effects of 
intervciHng treatments. The very low pre-post correlation for the Imitative Action subtest may be 
explainedvby the presence of a ceiling effect. Near-perfect and perfect scores were frequent. On 
the pretest^children averaged 4.02 poihts out of a maximum possible of 5,00, while the posttest 
mean was 4.34. , ^ • 

Table II- 1 also reveals,subtest-total correlations between each subtest and the pretest McCarthy 
summary score. These correlations have been corrected for the contribution of the subtest score 
to the summary score. The subtest-total correlations range from a low of .05 for Imitative Aition 



TABLE 11 1 

McCarthy test-retest reliabilities and subtest-total correlations 



Pictorial Memory 
Verbal Memory 
Conceptual Grouping 
Counting & Sorting 
imitative Action 
Drawing 

Number Questions 
Word Knowledge 
Verbal Fluency 
Opposite Analogies 

McCarthy Summary Score 



Test-retest 
correlations 



.34 
.61 
.78 
.62 
.18 
.75 
.42 
.57 
.66 
.61 

.87 



Corrected subtest-total 
correlations 



.32 
.61 
.70 
.72 
.05 
.64 
.39 
.69 
.66 
.57 



Note.-Rcliabilitics and subtesMotal correlation .are based on 64 children who received both pre- and 
post-McCarthy (52 TEEM and 12 Comparison children). 
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to :72 (or Counting and Sorting, with an average subtest*totat correlation of .54, The correlation 
invol^g the fomer subtest ii^ again haifipercd by ceiling effeots^ tiij'Jt is not unexpected that 
Imitative' Action Would! show tittle commonatity with the genetf^^ more cognitive abilities 
tapped by the other McCirtfiy subtests. The subtest-totai dbrrelation^; with the exception of that , 
for Imitatiye Action^ revea^t^ degree ^f co^^mpn ITariance ii^ong the subtests despite the 
diversity of abilities tapped,. ^' 

Pretest Date. o \ 

, Seventy-seven chi^ren ^pic indi^^doally tested on selected McCarthy Subtests in the fourth • 
wetk .qf September, \1<)7 1, Jh6se\te8ted:^ represented approximately half (N«62) ihe TEEM 
chiidrenr^domly selected from ench^of the six programs classrooms. In addition, one of the 
two Comparison, classes were tested in its entirety (N«15)r A second Comparison classroom 
became available for testing at a later date, and wo^sampled at posttest only:- ! , 

Data -analysis revealed that the CpinMso^ children scored consistently 'hi|rt«er across all 
McCarthy subtests on the pwt^st Jhih did TEEM chiltJitn (Table II-2).. Four of.these 10 mean 
differences achieve statistical ;Sigoin&nce/^tavor of Comparisons: Drawing, Number Questions, 
Word Knowledge, and Verbal Fluency^^ ,11 " , 

\ ' • TABLEIW ' . . 

McCarthy PRETEST MEANS AND F RATIOS FOR TEEM AND 
COMPARISON HEAD ^ART CHILDREN 



\ 

v. 


TEEM 
N«62 ^, 


Comparison 
N«15 


. -4. ■■ 

Test of 
Difference 




Raw Score 


X 

Raw Score 


F>Ratio 


INTEL^EaUAL SKILLS 








Pictorial Memory 


4.43 


5.67 


3.72 


Verbal (j(!emory 


17.89 


21.93 • 


1.94 


Conceptual Grouping 

^ 


5.63 


6.20 


1.24 


SOCIETAL A^TS AND SKILLS . 








Counting and Sorting 


5.58 


6.27 


1.07 


Imitative Action 


4.10 


3.87 


3.15 


Drawing 


6.39 \ 


^8.80 


4.84* 


Number Questions 


3.4fe . \ 


4.07 


5.14* 


LANGUAGE COMPETENCE 








Word Knowledge - 


13.58 


16.13 


' ' ' 4.87* 


Verbal Fluency 


6,52 


9.20 


4.32* 


Opposite Analogies 


3.32 


3.80 


1.25 



♦pcOS. 



As is often the case in quasi-experimental designs, the Comparison group was clearly not drawn 
from a population comparable to the TEEM children. The striking dissimilarity in pretest 
performance existed before any differential treatments were instituted. % 

Analysis of family demo^rafphic variables revealed no statistically signiflcant socioeconomic 
differences between TEEM children and the more able Comparison pupils (Appendix A), though 
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Comparison children were slighlly higher in mean family income and rate of employment and 
e4ucatk)n of head of household. No doubt more important in accounting for iht superior preidst 
preformance of Comparison sttidents was the fact that Comparisons were^ oii the a^rage; six 
weeks older than TEEM cliildren. /' j 

Several solutions to the pix>blem of the aoncomparable control groi^p were '>i>nsidered. 
Matching of Comparison childreq to selected experimental subjects on the^asis of pretest score;s 
was not feasible due to the lack of overiap between the: two pretest distributions. Of the 64 
pretested subjects who were also available at the ttme^ of the posttest, 24 of the S2 TEEM 
children and only one of the 12 Comt)arisons (46 and S9o of the two gibups, respective ly) scored 
below 67 (McCarthy summaiy score, pretest). Conversely^ the uppidr ranges, four CdmpaVisons 
(33%) but only one TEEM wbject (2%) scored above IQ5. . / ' I 

Analysis of covariSnce and gain score procedures were considered as post hoc methods of 
statistically equating the groups. / / 1 ; 

Analysis of Covariance / ^ \ I 

Of the 12 Comparison and 52 TEEM pupils who/received 1)oth pre- and post- McCarthy tests, 
the Comparisons were signiHcantly higher than TEEM subjects on the pretest nummary score 
(means of 89.17 and 67,84 respectively; F«8.24, pcOl). Although the difference in summary 
totals between the^ two groups was reduced, Comparitons maintained their superiority on 
posttesting (means of 116.08 and 98,56, F«4,45,p<-05). , ^ 

Analysis of covariance with pretest aflhe covariate^d posttest the dependent variable in a 
situation matrix where Comparisons htye higher pretest scores results in undercorrection for 
pretest din'erences, an4 makes the experimehtkl treatment look xlamaging, even in a simulated 
.case-where there are adtually no treatment effects (Campbell a^d Eriebachen 1970, pp. 196-197). 
It is therefore espedany noteworthy that such i covariance procedure Oi; tm preset data 
resulted in a higher adjusted posttest mean on. McCarthy summary score for TEEM phildren than 
Comparisons, equating on pretest^ even though the difference was not significant (102,43 and 
99 33, respectively; cfi AppendDt B). 

Because of-tjie limitations of covariance and the impossibility pf matching, the analysis of 
j)rog3in.efiect*;wS carried out in terms of gain scores* the least questionable procedure in view 
dfth^resent circutnslances. 

Pre* to Posttest Gains 

Program effectiveness may be assessed in terms of the amount of improvement attained by the 
children between the tW(Mesting sessions. Further, such an analysis in terms of gain scores allows 
comparisons between control and experimental groups, despite initial pretest differences. 

On all subtests except Imitative Action, there were statistically significant gains made by the 
TEEM children from the pre- to positesting (Table 11-3). On most subtests the Comparisons also 
made significant gams (Table IM), though the small sample size for Comparisons prevented 
several sizeabie increases (e.g., Verbal Memory) from achieving statistical signiflcance. 

Table 11-5 indicates that on 70% of the subtests the TEEM children achieved a greater raw 
score increase from pre- to posttesting than did the Comparison chiklren. These subtests are 
Pictorial Memory, Verbal Memory, Conceptual Grouping, Counting and Sorting, Number 
(Questions* Word Knowledge and Opposite Analogies. On two of the seven subtests (Pictorial 
Memory and Word Knowledge) the differences in gain between the two^ups are statistically 
significant a^ pcOS. The tendency for greater absolute gains by TEEM children persists despite 
the fact that treatment effects relative to control group gains are usually underestimated by gain 
score procedures when controls show initial pretest superiority (Campoell and Eriebacher, 1970, 
pp. 197-198). 

Figures IM through 11-4 present the pretest (Fall) and posttest (Spring) raw scores on the- 
selected McCarthy subtests for both the Comparison and the TEEM children. The blackened 
sections of the graphs elucidate the gain scores, or the improvement in subtests scores from the 
pre- to the posttest. In only one instance (Figure 11*2) is a decrease in scores seen from the Fall to 
the Spring testing. The maximum possible score is indicated beneath each subtest title. The 
McCarthy Scale of Children's Abilities is designed for children from age 2Vt to age 854. Hence the 
appropriate normative score is a more valid criterion than is the maximum possible score. On all 
graphs the dashed lines present the normative scores for five year old children. This is somewhat 
misleading, however, since there is a difference in the group mean ages, with the Comparison 
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• TABLE 11.3 

PR6. AND POST^McCARTHY SUBTEST MEANS AND F RATIOS 
FOR TEEM HEAD START CHIIX.REN ' 



• / 


Prefesr 
N»52 


Posttest 
N«S2 




Test of 0 
Gain 


t 


Raw ^£*ihtw^ 
-H- 


i**w score 


Gain 


F-ntio 


INTELLEaUAL SKILLS 








— 


l^ictoriai Memory 


4.40 


S.«5 


♦ 1.45 


15.91 


Verbal Miemofy 


16.52 


22.37 


♦ 5.85 


24.51 


Conceptuaf Grouping 


157 


7.S0 


* 2.13 


56.99*' 


SOCIETAL ARTS AND SKILLS 










Counting and Sorting 


5.29 


^.06 


♦ 2.77 


66.63" 


Imitative Action 


4J)4 


- 4.29 


t 0.25 


1.93 ' 


* OrawSng • 


6.02 


J 4,85 

• 


♦ 8.83 


168.a9'' 


Numlw Ooeslions 


3.67 


4.96 


♦ !29 


41. 07** 


LANGUAGE COMPETENCE 










Word Knowledge 


13.25 


16.67 


* 3.42 


45.79" 


Verbal Fluency 


6.15. 


9,67 


* 3.52 


48.95" 


Opposite Anafogies 


3.13 


4.35 


♦ 1.22 


49.48** 



analyses of variance. 
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pcOL 
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TABLE 11-4 

PRC. AND POST McCarthy subtest means and f ratios 

FOR COMPARISON HEAD START CHILDREN 





rrciesi 
N«12 


Posited 
N»12 


uain 


Te*t-of 
Gain 


Jf 

Raw Scone 


% 

Raw Score 


F^rartb 


INTtJLLECTUAL SKILLS 










Pictorial Mcmofy 


6.G8 


5.75 


- ^33 




Verbal Memory 


22 58 


27,24 


• 466 


4.17 


Conceptual Cfoonmj^ 


6.92 . 


8.33 


» !4» 


6.48* 


SOCIETAL ARTS AND SKILLS 










Counting and Sorting 


6.58 


8 83 


♦ 225 


JO 40" 


Imitative Action 


3.92 


4.58 


♦ 066 


6 77* . 


Dravymg 


1008 


20 58 


♦10 50 


50 70* ♦ 


NumWr Quc^l»ons 


4 00- 


4 67 


♦ 067 


3 52 


LANGUAGE COMPETENCE 










Word Knowrfedge 


)6 75 


17 83 


♦ 108 




Vcftwl Fluency 


8 33 


!3 67 


♦ S 34 


30 12''' 


Opposite Analogies 


3 92 . 


4 58 


* 066 


2 84 . 



Noie.-Stattsi*caUy significant gains are evidenced by signiftcant inals rnain effects (pretest to posttest) m 
analyses of variance 

*p<05. 
♦•p<01 



TABLE li s 

RAW SCORE GAINS ON MCCARTHY SUBTESTS FROM FALL TO SPRING 
FOR COMPARISON AND TEEM CHILDREN 





MeinGairti 


Test of ' 
iriimemc in uiirs 


Comporhoo 


TEEM 
N«S2 

*f* 


F-mito 


WTELLECTUAL SKILLS 








Pictorial Memory 


- 0,33 


♦t,4S 




Vefbal Memory 


♦ 4 66 


♦5.85 


<J 


Cooc^ptUdf Grouping 


♦ K4! 


•2 13 


1 23 


SOCIETAL ARTS AND SKILLS 








CounUngJiAdSortmg 


* 2 25 


*2.77 


<t 


fmiuUvc Action 


♦ 066 


♦0 25 


1 1) 


Dtaw]r\g 


*10.50 


♦8 83 


I 12 


Ujtnb^f QuevtJOfts 


♦ 0.67 


♦JJS"^- 


288 


LANGUAGE COMPETENCE 








Wofd Krtowltdge 


* 1 08 


♦3 42 


3 89* 


Verbal FfucntV 


^ 5 34 


♦3 52 - 


2 SO 


Oppoytc Analogies 


♦ 066 


*t 22 


1 81 



Note -Statisticaliv ^jgTMficani diffetences between gaim of TEEM and Comoa/ison children a/c ev»defK:ed by 
si^ificant yoypv .CnmparJson Enper^meniai) by trials fp^e to post McQfthy) mwyactiom m the repeated 



?nea-Sores an^lyy^s vartance 
•pcOS 
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gfojip being six weeks older than the TEEM children Appendu C reveals that a dissimilarity m 
age at the time of the piostt^ is particularly worthy of notice NorroatWely v there is a greater 
increase in raw scores for the six month period between 4V4 and 5 than 4 and 4V^. The six week 
age variance is, henci^, more crucial when analyzing the fK>stte$t than the pretest data^ 

Figure IM presents the pretest and gain sconjs attained on the three genera! goal areas of 
inteltecttsal Skitls. Socieial Am and Skills, and Unff^age Cdmp<frence These raw score figures 
were determined by the addition of the sibtest raw scores subsumed under each of these three 
general variables. Although the Comparison subjects began the school year with a hi^er mean 
raw score and ended the year closer to or above the norm, the TEEM children gained more in 
Intellectual Skills and in Un:;uage Competence* by 3 6, and 1.1 pomts respectively. On the 
Societal. Arts and Sitills clusie:: ibe Comparison children gained seven-tenths (0.7) of a point more 
tuan the TEEM children. 

Figure II-2 illustrateb the raw scores for subtests included in the ^neral category of 
Intellectual Skills* Refer to Figure 12 for a detailed account of the content of the various 
subtests. 

The Comparison group decreased from the pre- to the posttesting on the PictonaJ Memor>- 
subtest. Conscquctitly» their posttest score is below that of the VEEU chiWren althou^ their 
pretest score is higher. On all of the subtests included in the Intellectual Skills variable the 
Compar^n children's pretest score was higher but the TEEM children gained naore from the pre- 
to the posttesting. Neither group achieved the normathre score on any of the subtests 

The data for the subtests which comprise the evaluation of Societal Arts and Skills are shown 
m. Figure II-«. On all four of these subtests^ except Imitative Action, the Cqmparison children's 
pretest score was higher thAn the TEEM children's. Gains were greater for the latter group on 
Counting and Sorting and on Number Questions, and TEEM students more than doubled their 
pretest score on Drawing. The Actual numeric^, gain by the Comparison group on this subtest was 
greater, as was their gain on Imitative Action The Companson group met the norms for 
ftve-year-olds on the Drawing subtest. 

The Language Ccmipeten^^ variable included three subtests, and these scores are presented in 
Figun; tM On all of Ufesc ;>uoiests the Comparison chiWren began at a higher level than TEEM 
children, and on two of these Ihret the TEEM group made higher gains The Companson jyoup 
me! the normative mean on the Verbal Fluency subtest, and both groups were witbin one half 
(0 5) of a point of meeting the norm on the Opposite Analogies subtest 

Analyses were run to determine if there were differential gains Uom pre- to post-McCarthy by 
sex of the children No such interactions were evident in terms of McCarthy summary score or 
the three TEEM goal areas of Intellectual Skills^ Societal Arts and Skills, or Language 
Competence^ One statistically significant interaction between sex and gams occuned on the 
Conceptual Grouping subieit (T- 10.97, p< 01) This may be explained by the fact that 
independent of the companson-expenmeniai Qistmctson. females (N=29) gamed an average of 2 9 
\Knnis on the subtest pre* to post, while males (N-35) gained only 1 3 raw score points^ 

The coordinator of Head Start pix>grams for the Lincoln area rated the expenmental 
classroom teachers on their implementation of the TEEM Program (Table II-6) A scale of i-9 
was usedv the possible ratings ranging from '^barely atl^ptabje*' through '^average" io 
"outstanding" The highest ratings reflect thrprogram coordmaior^s judgement that the teacher 
was successful in translating the TEEM approach in to. classroom activiiief and attitudes 

The ratings were used to rank classes on TEEM implementation. These ranks wea* then 
correlaied with the ranks assigned the classes by assessment of the average gain achieved by 
children from pre- to posttesting on the McCarthy summary score (Table 11-6) The latter data 
was not known to the program director who rated teachers The rank-order correlation of 63 
reveals a notewonhy relationship between t^uccessfu! TEEM implementation and children's 
cognitive gains as measured by tht: individual:) -administered. $iandafd»?cd test (Spo^rmin 
r^nkKlifference method p- 6?) 

Solomon Fouf-Group Design 

All childrt-n pre^nt m the six TEEM and ?wo Companson classrocn^^ wc?c fcsttd m mid-Apn! 
Thfe resulted m approximately iwjcj; as majoy chiidrvn (99 TEEM and 29Companson> 
wiih vahd posttest -data on the McCarthy than with pretest 

The mam effecl of the pretesting experience on posttest scores was determined by u.. a 
Sol'vnon Four-Group DeiMgn (Campbell and Stanley, 1967 pp 24-25) Companson and Tl 1 M 
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TABLE U-6 

RELATIONSHiP BETWEEN TEEM TEACHER RATINGS 
AND CHtLpREN-S COGNITIVE GAINS 





TEEM lit^>kmcmatiOf^: 
Teacher Rating Ass^ied 
by Profram Coordkmof 


Classes Rjfiked by 
Teacher Effective^ 
ness as Rated 


Oasses Ranked by Children's Mean Gain 
on McCarthy toial, pre* to posnesting 


School A 


• 6 , 




2 


School B 


6 


^3 


3 


School C 


5 


3 


1 


School 0 


3 


4J 


5 


School E 


3 


4J 


6 


School F 


2 


6 


4 



P<ote.-Rank-differejKe corrtlatton between last two columns. p=.63. 



f 

"subjects who received pretests (groups *one and two for this analysis) were not signifkantiy 
different on either individual sublet or McCarthy sunwnary posn^jst scores from Comparisons 
and TEEM children who received pos^est oniy (group three and four). Nor was there an 

^ interaction between the pretesting factor and the control^xperunental distinction. Therefore the 
^ect of pretesting perseo^ the postte^ performance was consdered null 

Posttcst Data 

The ^diuon of the second Comparison cJass at posttesting yielded a composite group (N-29) 
which was mote comparable to posHested TEEM children (N^) than was the case for the 
smaller pretested samples. For example. Appendix A revcab that demc^phic data on the total 
Comparison and experimental groups is less divergent than was true at pretest. More important, 
the doubling of sample size at posttest resulted in an identical age of children in the posttest 
Comi^son and posnest TEEM groups, (Both groups average four years, 10,8-months at time of 
posttest K 

Table II-7 indicates that TEEM children are slightly superior on seven of the 1 0 (70%) McCarthy 
subtests at posttest. These are Pictorial Memory. Conceptual Grouping. Counting and Sorting, 
imitative Action. Number Questions, Word Knowledge and Oppoate AnaiogieSw On one of these 
(Number (>ie$tions> there is a statistically significant diffiprence in favor of TEEM chiklren at 
p<,05. The fact that Comparisons scored higher than TEEM ddWren on all 10 of the pretest 
subtests, but TEEM surpassed Comparisons on seven of the 10 post-subtests, is certainly 
noteworthy despite rjie lack of statistically significant differences. 

Means on the total posttest sample for the three TEEM goal areas of Intellectual Skills. 
Societal Arts and Skills, and L^pgua^ Competence are virtually identical for the TEEM and 
Comparison groups, as Table 11-7 indicates. Similarly, posttest McCarthy sunmury scores are very 
smiilar for TEEM children and Comparisons ^means of 97.83 and 98.07, respectively) 



Motivational Base 

The umruroent used in the assesanent of the children's motivational base growth was the 
Sch^ffer Behavior Inventory (SBI). (For a more detailed description of SBI. see page 7.) 

As with any rating scale, the rehabihty and validity of this inventory of ciassro<Hn behaviors is 
dependent lar^ly on the child raters (in this case, teachers) In three of the six TEEM Head Stan 
classes the teachers changed during the school year. Hence, nine different raters were mvolved m 
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TABLE li-7 

McCarthy posttest means and f^tkk for 
teem and cok^arison head start children 



• 


TEEM 
N»99 


Gxnpartson 
N«29 


Tot of 




V 
A 

Raw Score 


X 

Raw Score 


F-fatio 


INTEL-LECTUAL SKILLS 


41.23 




1 


PtcioriaJ Mcrnory 


5.96 


5.14 








22.66 




CbfKeptuaJ Gfoupmg 


. 7.69 


7.55 


<1 


SOCIETAL ARTS AND SKILLS • 


32.13 




1 
1 


Counting ^Sgrdng 


8.12 


8.03 


<1 


Mutative Action 


4.28 


4.21 


<1 


Drawing 


14.85 


15.49 


<1 


Number Questions 


4.88 


4.17 


4.48* 


LA.NGUAGE COMPETENCE 


30J5 


30.72 


1 


Word Knowledge ^ . 


16.39 


I6'l0 ■ 


<1 


VerfeAl Fluency 


9.79 


10.52 


<1 


Opposite Analogies 


4.36 


4.10 


<1 



•pc05. 



rating the TEEM children. Chahi^ in the raters leave the data, especially the gain scores, in a 
suspect position. The most valid pre-post data therefore comes from the three TEEM classes 
where there no teacher change. The same teachers completed the SBI protocols in the Fall 
and in the Spring. This is an important factor because individual differences in operational 
definitions of such subjective terras as '^likes,** "angry/* **eamestly," •*enjoys" wiU yield 
differences in ratings. Although individual differences among the three teachers who did not 
change from the pre- to posteditings are expected, it would not be anticipated that their 
definitions of the behavior over the year would draSScally change. The conjecture t^at these 
teachers may have utilized the inventory's points of emphasis as a working fraiD^- ork for 
dassroom behavioral analyses during the school year is not unlikely. 

The ratings are averaged by groups, some groups having more than one rater at different times. 
Differential ratings may result if the judgments were based on a six month, or a six week, sample 
of the chiWren^s behavior. Consequently, another factor to be considered is the length of time 
the new rater had been the classroom teacher. 

Psydiometncians and social psychologists acknowledge that this Likert-type of rating scale is 
notof^usiy subjective. One prevsdent artifact is' Experimenter (Rater) bias. Inventory results 
peneraUy reveal the rater's own individual preferences and prpjudices about what such a test 
should reveal. Another artifact of the Likert scale is response set^r response bias. For example, a 
rater may fall into a middle-of-the-road set which prevails throughout his ratings ahnost r^ardless 
of the item content. The facts presented in the preceding three paragraphs shouW be considered 
as one analyzes the SB! data. ' 

In view of these less than optimal methodok)gical points, the reliability data is quite good. The 
reliabilities of the items on the SBI from pre- to posttest ing are superior to those usually obtained 
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for individual items. This is even more noteworthy when consideration is given to the facts of a 
seven^onth span between testing sessions and of mid-year teacher changes in three of the six 
classrooms. The range of the reliabiUties oCthe IS individual items for all six TEEM classes is 
between .40 to ,59, with a mean reliability of .50. Considering just the three classes which 
experienced no teacher change for the school year, the item reliabilities range from .48 to ,72, 
with a mean of .63, In the interest of clarity, the six combined TEEM classes will be referred \o 
in this data analysis^ as group A and the three TEEM classes where the teachers did not change in 
mki-year as group B. 

A summing of the items in the three m^'or variables of Task Onentation, Extraversion and 
Hostility yields the subtotal reliabilities presented in Table II-8. 



TABLE 114 

SCHAEFFER BEHAVIOR INVENTORY 1 EST-RETEST RELIABILITIES 





Croup A: 
Children from M six TEEM 
classes 
N»82 


^ Group B: 
Children from>three TEEM 
classes with no t^icher change 
N«36 


Task Ortenutfon 


.63 


.76 


Extraversion 


.53 


.69 


Hostility 


.58 


.67 



Gain scores reveal numerical improvement from one testing session to another. The SBI gain 
scores for both group A and group B arc presented in Appendix D, and are illustrated in Figures 
11-5 through 11^. Figure II-5 presented the pre- and posttest mean ratings received by the 
children on the five items included under the Task Orientation rubric. The teachers judged their 
children as being average ("half the time") or above ("frequently") on these items. Gain scores 
arc statistically significant at p<c05 level for group A on two items: "Stays with a job until he 
finishes it " and "Becomes very absorbed in what he is doing." Consistently group B ratings are 
slightly higher. Decreases from the pre- to pdsttesting are seen for this group on two items: 
"Works earnestly at his classworic. Doesn*t take it lightly/' and "Watches cai^efully when a 
teacher or classmate is showing how to do something*" 

The'ratings on five items clustered under the Extraversion variable are disclosed in Figure 11-6. 
Overall, these judgments are somewhat higher than those of the previous figure. Group A 
experienced virtually no change from the pre- to posttesting on three items: "Likes to take part 
in activities with^^ers." "Eiqoys beii« with others." and **Secks sodal contact with others " 
The change on one item: *Tries to be with another person or group of people," was statistically 
significant at pc.05 level, as was the rating change for group B on one item: "Likes to take part in 
activities with others. " 

Figure II-7 presents the mean teacher judgments on the Hostility items. As would be expected, 
the compoate picture is one of lowered ratings, in the range of "never" to "occasionally," and 
negative gain scores. Both groups achieved the latter for one item: "Slow to forgive when 
offended." In addition, group B ratings decreased from pre- to posttesting on two other items: 
"Gets impatient or unpteasant if he can^t get what he wants when he wants it," and "Angry when 
he has to wait his turn or share .with others.** The two rating sessions yielded virtually no change 
for group B on one item: "Complains or whines if he can't get his own way/* 

Overall analysis by item indicates three statistically ^ificant gain scores (p<.OS)/or group A 
and only one for group B. On an analysis of fifteen items the latter finding is not superior to 
what would be expected merely by chance. On seven of the 15 items for group A and on nine of 
the 15 Items for group B, the pre* to posttesting changes were in the appropriate direction (e.g., 
increases for Task Orientation and Extraversion items, and decreases for Hostility items). 
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Trends ofi the Ihree miuor variables of Task Orientation, Extraversion and Hostility are shown 
in Figure 11-8. For both groups the means judgments are positive, with Task Orientation and 
Extraversion ratings ranging from **half the time** to 'Trequenttyr and Ho^itity ratings ranging 
from "occasionally** to '^almost never/* The direction of change is appropriate^ln twp of these 
three categories for group' A« Overall improvement from pre- to posttesting on all three clusters is 
seen in group A ratings. However* such an increase is desirable only for Tadc Orientation and 
Extraversion variables. Group B ratings did net appreciably change on Task Orientation and 
Hostility but did improve on Extraversion, - 

The two Comparison class teachers did not complete the SBl on their pupils ia the fall 
Therefore the only TEEM vs. Comparison group data available are on the posttest. The group 
means of these ratings for the TEEM and the Comparison children arc presented in Appendix E. 
The mean ratings on the three basic behaviors arc revealed in Table 11-9. 

TABLE 119 ^ , 

SCH AEFFER BEHAVIOR INVENTORY POSTTEST MEANS FOR TEEM 
AND COMPARISON HEAD START CHfLOREN 





TEEM 


Comparhon 




. N«104 


N»26 


Task Orientation 


4.6 


4.5 


Exuavmion 


4.8 


4.7 


Hostility 


3.1 


3.4 



There were no statistically significant differences between the groups on either the items or on 
the subtotals of the three major variables* The item ratings for the two groups were quite close, 
with a negligible rating difTcrcnce, A comparison of the TEEM and Comparison group ratings 
presented in TaMe 11-9 indicates a sli;^t superiority In )he dedred direction in favor of the TEEM 
children^on alMhree variables (Task Orientation, Extraversion and Hostility). 

Gassroom Process , ^ 

Information on classtoom process was'gathered through observation techniques. Two different 
instruments were utilized in this phase. One was an experimentally developed technique designed 
to assess certain process goal? of the TEEM program. The Classroom Attitude Observation 
-System (CAOSi was a combination of observation instrumetits which included categories for 
activity types, incidence of inappropriate behavior, and grouping strategies. It includes an 
experimental modification in which the teacher is absent from the classroom with observation 
ongoing. The other instrument was a more traditional observation format develCped by Stanford 
Research Institute as part of their national sfudy of Head Start Planned Variation. The Classroom 
Observation Procedure (COP) has categories for physical plant quality, activity types, as well as 
five-minute interaction sequences. The analysis will incorporate a validity matrixin which 
information gathered by these two techniques will be compared: 

The Classroom Attitude Observation Schedule data Were reduced into summary. variable<i for 
analysis. These variables focussed on key concerns of TEEM program implementation. The initial 
analysis looked at mean group size, which was calculated by dividing totai number of children 
counted in a twelve-minute phase by the number of groups counted in that phase. A second 
variable was that of mean number of children engaged in an activity at any one time during the 
tweive*minute phase. In terms of the recording anstrumenf« this was mean cell size and was 
calculated by dividing the totai number of children counted in a phase by number of cells used in 
a phase. The third variable wos inappropriate behavior This was simply a total count of incidents 
of inappropriate behavior observwl during the twelvc-minute phase* Two other variables tested 
were mean number of groups in a phase, and mean number of activities for a given phase. 

The above five variables were combined for TEEM classrooms^ amfd Comparison classrooms, 
and the group means over the 3 phases were analyzed with analysis of variance procedure. Results 
of these analyses appear in Table IMO. 
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ANALYSES OF VARIANCE OF TEEM AND COMPARISON CLASSROOMS ACROSS PHASES 
FOR SELECTED CAOS OBSERVATION SUMMARY VARIABLES 



V«iabie 


1 

Source of Variation 




ss 


MS 


F 


?nc«n v>*oup >'<c 


&oup (Teem vs Comparfson) 


? 


0006 


0006 


<1 




Phase 


2 


0340 


0170 


38?* 




Group X Phase 


2, 


0076 


0.038 


<1 


BehavKX 


G<oup (TEEM vs Comparivonj 


1 


193 39 


193 39 


20 80'* 




Phase 


2 


27975 


13987 


14.36*** 




&oup X Pha&e 


^ 2 


262J0 


131 35 


1348* 


Mean Oil Sue 


^io signsf'icam, results «^ere obtair>ed 


Mean Numb^ of 
Gfoups 


No s<gn«ucani results vrtre obuir>ed 


Mean Ntunber of 
Cdfs 


^io Significant results were obtained 




' i 

-J 



nxlO 
•pc05 
**p<01 
•••p<.001 
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Another analysis used independent ratings of teachers. The Lmcoln Head Start Director was 
asked to rate Jl of Ihe teachers on level of implementation (c,f. page 12). The lowest teacher 
rated was a 2" aisd the twtf'highest were "6V\ Sir^ these ratings were assigned independently 
oi u.c vanables examined for CAOS.'one high-rated teacher was selected from the two high rated 
feachers^ and was compared with the low rattfd teacher on variables showing variation on CAOS 
i-or further examination on the data, activity categories were colJaj^sed into five broad types of 
activities. Type I consists of "traditional" academic activiUe* and includes arithmeUc. tanguage, 
social studies and science. Type II consists of cogniUve learning activiUes for young children bu! 
Mot mahe "traditional" sense. Included in this type are stories, games, puzzles, arts, crafts 
cooking, and building. Type III consists of play and role playing. Type IV consists of snacks and 
management. Example of management activities are cleaning up. bandmg out material going 
after supply, sharpening pencils, teacher and child talking. These activities ^ppfy to both 
children and adults. Type V is non-focused actmttes. such as tr»nsjt»ona» actmUei. children out 
of the rcK>m. and wandering. 

Tpese five activity classes were used to examine the porportion of Ume ^t by adults and 
children m each activity type dunng phases A and C (when both were m the room together) 
pese data wea- examined for the same high-rated classroom and low-rated classroom (Fimres 
u-9andll-10) 

Due to the exploratory nature of the research and the small sample of clasvooms generous 
confidence hmits were set for acceptance of statistically significant findings The maximum ievcl 

^Ol thr V*>o highest rated tcachcn, one Uagfct ihe whoie year in TEEM He*d Sun *hiJc the jeconJ »ai i 
.Kid year fepUcemcnf The forme? teacher *3S uUlue»3 >n the analysi* o( the insp^etDrrHatjon outcoRvrt 



acceptable was set at 10 It was felt that this IcmM N*ould gjvc a suiffCient induction that 3 
vanabJe should be explored in future research 

Of the five summary variables tested for analysis ofv^nance two Ucrnonstraied significance at 
acceptable levels {Table U- 1 0) 

Mean group uze over al! et^.t classrooms imitcated differences aaoss groups were significant 
(p<JO) There were no ssgmficaht difference between T!:EM implementated vla^ses and 
Comparison classes 

The roost conspicuous different appeared with tmppropruxte t>ehavior Analysis of this 
variable indicated that all ^Durces pf variance werC'Significm! (groups p< OL phases p< 001 . 
group by phase ^05). Figure IMl graphically describes this variable 

In companng high and low rated classes* with only one classroom m each category . inappropriate 
behanor was no! tested for statistical sigmficance. 'However, a chart was prepared parallel to that 
for the summar>' vanable* and the interaction pattern runs an essentially similar course (Figure 
IM2). . , 

When adult partidpatibn was compared to child partiapation in the high <tnd low classrooms* 
rank order correlation between aduh presence and child presence in the vanous activity types was 
-21 in the low rated dassroom (Figure IMO). In the high rated clas»oom. the correlation 
between adult and child participation was .80 (Figure IM i ) 

The G^ioom CMiservation Procedure (COP) was designed by SRI to ( 1 ) assess the degree of 
implementation of dassixxmi processes within various instructional models, and (2) assess chdd 
outcomes from the varying systems. 

To achieve th^ assessment goals, the. instrument is divided into thr&e sections Phy^cal 
Enviromnent Information, Classroom Cheddist* and Five4(tnute Observation. In the initial 
section, physical plant variables, such as lifting, noise teve! and seating arrangements are 
recorded. The second section entails a recording of w*&at each adult and child in the classroom is 
doing, yielding a '^snapsbot^ of the classroom. Several of these '^data pictures'^ would then reveal 
the group size* adult focus, and activities typical in that class In the third section the observer 
focuses on one individual* or group, and records in detail the "who'\ **to whom''. ^vhat*\ and 
**how** of classroom interaction. 

SRi-tramed staff recorded a minimum of 24 observations per class m five Lincoln TEENf 
classrooms. Normative data on this mstrumcnt is not presently available Consequently, to 
faalitate immediate utilization of the TEEM classroom observation data procured from SRIw the 
same observer was hired to make an equal number of observations m the two Lincoln 
Comparison classrooms 

Following the direction provided by the list of variables developed by SRI. the frequencies of 
35 separate variables were cx>mpiled from the classroom observation data For daU analysis and 
graphic presentation purposes, these variables were grouped mto the following vanabfe 
constructs 

adult instruction 

child -mitiated learning 

adult attention^five children or less 

child se!f-expressK)n 

child-chik) interactions 

positive behavior 

negative behavior 

A complete list of the COP vanables included m these vanable Constructs are presented m 
Appendix F CJenerally. both child and teacher behavior is included in a variable construct unless 
the variable construct title indicates otherwise 

In rhe interest of brevity, the data from only three of the ieven classrooms are presented Of 
thf^tw^-^ TEEM classrooms, those rat^tl the highest and lowest by the Head Star\ Director of 
the teachers* success in implementation of TEEM (seepage 12). and one is a Comparison 
classroom. In the m^onty of the varuble constructs, the r»^nd for the two TET.M da^ooms is 
the same, and is different from the Comparison classroom Appendix G provides the numcncai 
frequenaes for the three ciassrt>oms on the eight variable constructs 

Figure 11-13 depicts histograms on the two v-anable constructs included m the Cognitive 
category Both TEEM classes have a lower freq^ucficy over 24 observations of adult instruction, 
and a higher frequency of child*initiatcd learning than the Comparison class 
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Figure IM4 presents tmiividual histograms for the six variable constructs included in the 
Socio-AfTective categcwy. OveraO, the trends indicated that in the TEEM ciassroom theie was a 
lower frequency of aduh attentjon-Hve children or less, m<m child $elf*expies$ion, more single 
child-adult and m<Ke cfaild<hi)d inte raction s, more positive, and less Mgative behavior. The 
results of Chi square analyses between TEEM High and the Comparison group, and TEEM Low 
and the Cpmparison group are Igted ,in Table fMl, In the former analysis. 4 of the 8 
' comparisons arc sgnifkant (gK-Ol).- In the second analysis, 6 of the 8 of the comparisons are 
significant (2 of these ^05 ; 4 of these p<.OI ). 

Figures IMS and IM6 illustrati^ a profile of the same variable constructs in the Cognitive and 
in the.Sodo-Affective categories /or the thrw cbssrooms. The wide range of frequencies (see 
Appendix G) necessitated the ii$c of two- and three<ycie semi4ogarithraic presentation. When 
examining these two sununary figures, it is well to keep in mind that this type of di^lay 
minimizes difference's m higi^er frequencies and maximizes those in the lower frequencies. Also, 
connecting linerbetween points representing the variable construct frequencies are drawn only to 
present a summary profile/and are not intended to indicate a continuous variable oh the abcissa. 

Even wirh the present a^wence of normative daU^these data {msented reflect the usefulness of 
this instrument in determining objectively the ongoing processes within a classroom. 
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Fig. IM4. Individual histograms of the Socio*Affective variable 
constnicis from the Classroom Observation Procedure dau. 
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rig. IM6, Summary profil^of Gassroom Observation Procedure varUble 
constructs included in the Socio^Affectsve category. 



DISCUSSION 



The foUowing discussion' will focus on the sUted goals of this study. These, goals were 
concerned with the development of sponsor-specific outcome evaluation measures.. In addition, 
the data were submitted to mMHlirectional statistical analyses test for differences between the 
group of children in the Planned Variation TEEM Head Start classrooms, and a Comparison group 
in locally implemented Head Start classrooms. 

One goal of this study was the elaboration of an observational-evaluation system for the 
assessment of process goals within the TEEM classroom. This goal was met with the development 
of the Gassroom Attitude Observation. Schedule (CAOS). The CAOS technique represented a 
meld of methods taken from the experimental laboratory studies of children with current 
methods of observing social interaction in the classrxxmi* The modiUcation of the classroom 
environment by removing all adults (other than the observer) provided a crucial test of the 
implicit control system within the Head Start classroom* 

The results from the CAOS observation 5«quence indicated quite striking differences between 
the TEEM classroom style and the locally implemented rtyle. These differences were manifested 
despite the fact that the teachers in the locally impleiiiented classrooms participated, at their 
discretion, in the TEEM training program offered through the Planned Variation funds. HoweverT 
these Comparison classrooms did not receive the supplementary equipment and activity fiinds 
provided through Harmed Variation participation. The conspicuous finding was the relationship 
between CAOS "teacher absent*' phase and the manifestation of "inappropriate behavior** 
sequences in young children. The data indicate that, in the TEEM classroom* chiWren are used to 
self management, and continued in their learning experiences vrith the teacher absent. In 
contrast, children in the Comparison classes manifested large increases in this dimension of 
non-adaptive behavior during teacher absence. 

What explanation can be offered for these findings? The pioneer work of White and Lippitt 
(I960) has been influential in setting the direction of the "open education'* movement in this 
country. Their research strategy was very similar to the one employed with CAOS, in which 
natural groups were unobtrusively observed, and information collected on'leadership style and 
group development. Their studies looked at the relationship between leadership styles in informal 
boys clubs^, and behavior patterns in participants. They found a strong relationship between 
democratic leadership style and self-directed behavior in the boys. In contrast, the autocratic 
leadership style produced boys who were more authoritarian and hostile. 

It would not be untenable to assume that these kind of response styles can have their origin in 
the early school experiences of children. The data collected with CAOS provide information on 
the impact of the open system of early education in effecting self-du ited, independent learning 
styles in young children. 

Another set of related CAOS findings concerned the activity cycles of children and teachers in 
two TEEM classrooms, one rated as "high-implemented" and other rated as "low-implemented*'. 
Early experience in the task of widespread field delivery of the TEEM indicated that a wide range 
of classroom systems evolved, some high quality, most average, and a few unacceptable. In fact, 
the earl*' findings of the SRI national evaluation study of Head Start indicated that there was 
often as much "program variance'* within model sponsor sites as between different sponsor sites. 
With this in mind, a clpser look was taken at the range of outcomes with the six TEEM 
classrooms. The "low-iniplemented** classroom showed adults predominantly concerned with 
management functions, >yhile the children were involved in play sequences. In contrast, the 
"high-implemented** classroom showed adults involved in child-learning activities, and the 
children were engaged in cognitive-learning sequences. With a limited lange of six classrooms, the 
strength of these differences must not be over-emphasized. However, the information does 
indicate that a rigorously applied evaluation system can find extensive amounts of information 
unavailable to a casual observer. 

Another pervasive question in the evaluation of early education systems has been the 
relationship between level of mo^el implementation and cognitive outcomes in children. The 
former construct refers to the fit between an idealized classroom (in this case, a TEEM 
classroom) and the range of classroom styles seen in the real-worid field sites. One of the first 
priorit'ies' in developing a better understanding of the TEEM system in Head Start was the 
specification of the characteristics of the classroom system.'^This task was undertaken by 
members of the field instructional staff during 1971 "at the request of the national evaluation 
contractor. These guidelines were utilized by the Director in Lincoln in her rating of the level of 



TEEM impiemenUtion. On a nine^point Likert scale, the range of rated dassrcoms was two 
through six. No classrooms were described as near ideal criterion, while one was rated near the 
bottom* 

The specification of a prcgram-valid set of child measures was another goal of this study. 
Relating the classroom mean gains on the McCarthy summary score to the implementation rating 
produced a rank-over correIatk>n of .63. With this small sample, this is not a significant 
relationship, but does indicate the direction for a future replication study. A conservative 
interpretation of this finding is that the conditions that facilitate cognitive growth in young 
children share some commoinlity with the expressed goals of the TEEM implementation system. 

Additional information on the relationship between classroom process and child outcomes was 
offered through utilization of the data base collected by Stanford Research Institute (SRI). 
Economic and time limitations prevented complete utilization of >his data source. However, 
information gathered on SRI's Oassroom Observation Procedure was processed in time to be 
included in this report. This instnunent shares some conunon information with the CAOS 
system, as well as unique data on the "actors" in the classroom and the tiature of their 
interaction. A sample of only six classrooms does not permit direct statistical comparison of 
these two sets of information* However, a visual scan of the tables presenting these two data sets 
indicates some commonalities. 

One of these relationships indicates that the occurrence of Single Child-Adult interaction was 
highest in the TEEM^iigh rated class, followed by lEEM^ow rated and Comparison. This Hnding 
relates to the CAOS data whidi showed teacher and chiWrcn leammg togetiier in the TEEM-high 
rated classroom. As well, Uie incidence of ChikK^hild interaction was highest in TEEM-high rated 
followed by TEEM*low rated and Compaiison* 

, These similar findings, collected on two different observation instruments by different research 
groups, provide concurrent validity information on both the instruments and on the operation of 
the TEEM classroom. These data are currentiy being submitted to further analyses with the 
''teacherK}hild interaction^* data base to explore these reiatidaships. 

Another goal of this study was the development of a set of child outcome measures that were 
congruent with the overall goal structure of the TEEM program. A m^jor problem experienced in 
the SRI national evaluation study of Planned Variation, has been the deification of a test 
battery that is economical, valid and representative of the wide, range of sponsor goals. Some 
sponsors rest content with the Stanford-Binet Intelligence Test or the Wide Range Achievement 
Test as an acceptable 4utcoxr(e measure. Because of the complex goal structure of the TEEM, no 
single available test wai con^dered appropriate as an outcome measure This led to thestrateg>' 
of selecting different measurement tools for assessing the rnultiple outcomes of TEEM. The m^or 
source for the child data gathered in this study consisted of selected subtests from the recently 
published McCarthy Scale of Children's Abilities. In addition, teacher ratings were collected to 
assess children's growth in the motivational base area. 

In terms of their actractiveness to children, this selected battery proved very successful 
Children were stimulated by the variety of items, and none of the tasks presented serious entry- 
problems to these Head children. The measurement properties of the tests were adequate, 
except for one subtest in which many subjects topped out (Imitation). The results indicated 
differential gain patterns between the TEEM and Comparison groups. In two of ten instances, 
these §ains,were statistically significant, and indicated greater growth for TEEM children. This 
finding, combined with the previously mentioned relationship between TEEM implementation 
and child gains, gives evidence for the validity of this battery- for assessing TEEM outcomes. 
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Summary 

The most imporunt questiop facing thedccisiothmakcrsatthc Oflioc of Child Dcvclopmcnc 
conoerm the efTicacy of the additional fundi provided throt^b the Planned Variation study. In 
terms of deiimy of the TEEM program in PV Head Start, these funds were foamed in the 
teacher training effort, as well as provision for additional equipment and «ipi*es in the 
classroom environment. The expKdt question then relates to whether these additional resource^, 
were justiOed in terms of demonstrated differences in the children, teachers and community who 
receive Head Start services. 

T^as study w^ expUcitly directed to chiW outcotn-s, and the mfonnation presented hert 
^ mdicatcd that a quite different system of dasHoom organization existed tn these two sets of 
dasffooms. In the TEEM classrooms, children were oriented toward more independent learnings 
sequences, and teachers partidpatei. in these exf^riences on an individualized basis. In contra^^, 
the Companson classes were typified by children in lar^ grt^up learning sequenct^^and tnere wa« 
evidence that more severe controls were exerted in classroom management. Similar information 
was prodiiced through two mdependent sets of daswoom obseivaUon daU. If the values of 
partidpatory democracy^ and the aOendant self-directed learning styles, arc important goals for 
young^ children, then some indication is given here that partidpation in a TEEM Head Start 
classroom contributes to that end. 

Another set of data was concerned with cognitive growth as measured by a selected battery of 
psychological tests. The Jata gave some indication that TEEM children manifested an accelerated 
growth curve (in terms of McCarthy summary gain scores) on these tests, as compared to the 
Ccmparison group. However, only in a few instances were these difTercnccs statistically 
agnificant. If these goals are vahied within the overall goal structure of the naUonal Head Start 
program, information herein indicates TEEM has the potential to ihfluence these outcomes 

The information indicated a wide ran^ of classroom style variance evoKes throi^ the TEEM 
delivery system. Fwther. the range of outcomes was related to the quahty of that 
implem^tation within the dasaoom. However, the short time frame of thb study did not permit 
further vexaminatic.i of methods to bring all classrooms into an acceptable range of 
tmplemd|nUtion quaUty . 

Lackiiig direct information on the development of teacher competence withtn the TEEM 
delivery system, or »he delivery of the parent invobement system, no comments can be made on 
the relationship of these aspects of TEEM delivery to child outcomes 

Future Directions 

The study reported here, wjth limited resources, provided a mmunum of solid findings and a 
raft of mformation to use in charting new research directions for TEEM. One direction is 
self-evident: classroom observation procedures proFVide the necessary tool for exploration of the 
important, but heretofore methodologically elusive, pro^ goals of the open classroom. New 
techniques, such as CAOS. permit investigalion of soao-cmotiona! outcomes that have been 
unpossible to pursue with traditional paper-and-penal techniques. 

Another avenue of research potential is to follow this group of children ^rough their school 
expenences within the Follow Through feeder system- This longitudinal design has been 
developed, but additional funds have not yet been allotted 

This study has touched the surface of a number of unportant issuer m the design of ev^Jua;lO^ 
^'Sterns for compensatory programs. The great mass of unanswered questions stdl remams as a 
contmuing challenge to the field. 
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APPtNWX A 
OeMOGAAPHIC DATA AHALVSIS 



Pmesi Sample Mcim and F^UtkH for 
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APPtNOIX t 

POSTTEST MEAN RATINGS ON THE SCHAEFFEft BEHAVIOR INVt^iTOR V 
ITEMS FOR TEEM AND COMPARISON GROUP CHttOREN 
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CLASSROOM OeSERVATfON PROCEDURE VARIABLI CONSTRUCTS 
GROt^PED BY COGMinvt AND SOCiaAFFECTm CATEGORiLS ' 
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APPENDIX G 

FREQUENCIES OF CLASSROOM OSSERVATKM PROCEOORE 
VARIABLE CONSTRUCTS FOR THREE CLASSROOMS 
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